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Evaluation of NASA Space Grant Consortia Programs 


Abstract 


The meaningful evaluation of the NASA Space Grant Consortium 
and Fellowship Programs must overcome unusual difficulties: 

a) the program, in its infancy, is undergoing dynamic change; 

b) the several state consortia and universities have widely 
divergent parochial goals that defy a uniform evaluative 
process; and c) the pilot-sized consortium programs require 
that the evaluative process be economical in human costs lest 
the process of evaluation comprise the effectiveness of the 
programs they are meant to assess. This paper represents an 
attempt to assess the context in which evaluation is to be 
conducted, the goals and limitations inherent to the 
evaluation, and to recommend appropriate guidelines for 
evaluat ion. 


Introduction 

The NASA Space Grant Program inaugurated in September 1989 
was designed to catalyze the development of ideas, programs, 
and a broad-based institutional commitment and infrastructure 
that will, in the long run, satisfy the following explicitly 
or implicitly stated objectives: 

1. To arouse the interest of a generation of K-12 students 
in mathematics and science, to improve their levels of 
competency in such subjects, and to stimulate their 
collective interests in, preparation for, and dedication 
to careers in diverse technologically-based disciplines. 

2. To arouse the interest of the general public in 
aerospace-related activities of NASA and other 
governmental and private agencies, to get John and Mary 
Q. Public to stop yawning at the day-to-day successes of 
NASA et al.. To develop a public appreciation for the 
scientific and technological challenges of aerospace 
science and technology, to develop and understanding of 
the scientific and technological benefits to accrue from 
a vigorous program of aerospace-related research and 
development, to develop a public understanding of the 
economic benefit of such programs to the nation, indeed, 
to convince the public that such programs are imperative 
to our economic health and national security. 

3. To engender broad-based pubic support and the associated 
political constituency necessary for budgetary 
commitments essential to realize these objectives. 
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4. To co-opt increasing devotion of resources from State, 
Federal, and private agencies toward aerospace-related 
research and development and human resource development. 

5. To assure a stream of well qualified and motivated 
technologically educated students being graduated at the 
BS, MS, and PhD levels, adequate to meet the needs of 
NASA, DOD, and our aerospace industries, and thereby 
preserve and enhance our technological competitiveness, 
balance of payments, national economy, and national 
security . 

6. Affirmative action goals to enhance the opportunities for 
affected minorities and women are an independent 
objective and inherent to and a necessary condition to 
the meeting of the above stated five goals. Given the 
demographics of the work force projected for the coming 
decades, even the most mean-spirited, socially 
retrograde, morally perverse, but intelligent individual, 
would adopt as a Machiavellian strategy, a strong 
pro-affirmative action bias. 

The above stated long-term objectives of the Space Grant 
Program define the context in which one can attempt 
evaluation of the National program and several State 
Consortia. The resources currently allocated to the task are 
woefully inadequate to fulfill the above goals but they can 
encourage the development of a cadre of committed people and 
institutions, and the establishment of effective means of 
communications among them. 


Goals of the Evaluation Process 

All that can be asked of the current programs at the current 
levels of funding commitment is the demonstration of 
promising approaches, and the identification of pitfalls, and 
promising looking but blind alleys, so that, when (not if) 
Congress, NASA, the States and private industry develop the 
resolve to provide the levels of investment necessary to 
attack problems that must be attacked we will do it with 
greater wisdom and efficiency. 

Thus, the primary purpose of the evaluation process must be 
to set the stage for a cost-effective scaleup of the 
operations of the Space Grant Program. Since significant 
institutional and individual stakes will be riding on these 
evaluative assessments they will be necessarily biased. 
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Limitations to the Evaluation Process 


Don Griffin, formerly of West inghouse ' s Bettis Atomic Power 
Labs articulated what I will call, Griffin’s Law : 

"Under the best of circumstances, the product of 
objectivity and expertise in any one observer is a 
constant . " 

This "law", somewhat reminiscent of Heisenberg's Uncertainty 
principle, articulated in the context of evaluation of high 
technology programs in a different discipline is applicable 
to NASA's goal of evaluation of the Space Grant Program. We 
will have to rely to a considerable extent on people with an 
"ax to grind" to prepare the evaluations. The best that we 
can do is to require that the bases for the evaluations be 
clearly articulated, that the underlying data be public, that 
the authors of the evaluations be identified, and that those 
responsible for reviewing the evaluations do so with clear 
understanding of the inherent biases of the authors. I am 
sufficiently sanguine with regard to human nature to trust to 
the basic intellectual integrity of the evaluators (ensemble 
average) not to fabricate the data. On the other hand, there 
will be wishful thinking that manana we will see the light at 
the end of the tunnel and our programs will be productive. 


NASA has already missed (I believe) the opportunity to 
perform the evaluations with scientific rigor. To do so they 
should have rank ordered all of the Space Grant Proposals and 
funded all of the odd-ranked proposals, denying funding to 
the even-ranked proposals. One could then compare the 
performances of paired States with universities of inherently 
comparable qualities and would-be Pi's of comparable 
imagination and enthusiasm. Such a controlled experiment 
would then allow one to isolate the effect of NASA funding on 
the outcomes. NASA was probably wise not to conduct such an 
experiment. The basic message is that we shall be hard 
pressed to measure the extent to which the NASA funding was 
the cause of the measurable advances. Those institutions and 
individuals represented at this meeting are aggressive, 
capable, and dedicated to the Space Grant goals. They would 
have found alternative ways to achieve some of the successes 
that we shall report. 


It should also be noted that 
one of many factors that will 
above stated goals. The over 
world economies, the national 
severity and importance of so 
geopolitical trends and Congr 
reactions to them, particular 
for DOD, NASA, and education, 
impacts on the very variables 
to assess the NASA Space Gran 


:he Space Grant Program is only 
affect the realization of the 
all state of the national and 
perception of the relative 
:ial problems, the worldwide 
ass' and State legislative 
Ly as they may affect funding 
can be expected to have major 
that one would like to evaluate 
: Program. 
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Conclusions and Recommendations 


The Space Grant Program is in it's infancy. We are just 
beginning on the learning curve. The resources allocated to 
the problems are at the proof-of-concept level. The several 
Consortia are starting from diverse positions, have 
established diverse initial strategies, have articulated 
diverse short-term and long-term goals, dictated by 
conditions parochial to their specific situations. 
Accordingly, the following criteria for evaluation are 
recommended : 

1. Quantitative Space Grant-wide objective functions should 
not be defined to evaluate individual programs. 

2. It will probably be useful to gather data on standard 
quantitative measure of productivity (enrollments, 
degrees granted, papers published, patents awarded...) 
to report for the NASA Space Grant Program at large. 

The data will be of most interest in terms of 
year-to-year changes. 

3. First year results should not be given heavy weight. 

The evaluations of the programs should be made over a 
longer haul. 

4. Significant experimentation with and modification of 
programs is anticipated in the early years of the 
programs. Evidence of internal evaluation and 
responsive adaptation of program strategies is to be 
encouraged. Wherever possible such evaluation 
processes should be designed into the programs to 
assure timely feedback. Such internal use of evaluation 
should be the primary purpose of Consortium evaluations. 

5. The consortia should be encouraged to develop 
parochially appropriate (that doesn't mean self-serving) 
evaluative criteria. 

6. The evaluative criteria and means of assessment should 
be anticipated to be dynamic in the early years of 
the program. 

7. From annual review of the individual criteria and 
evaluative processes will evolve a more systematic and 
common basis for evaluation as the programs mature. 

8. NASA should provide early general guidance for the 
manner in which evaluation issues are to be treated 
in the September annual report. 


13 



9. It is our understanding that the first step in the 
Consortium internal evaluation process is to review the 
NASA RFP goals and to restate them in Consortium- 
specific terms. 

10. During the formative years the primary thrust of the 
evaluation process is to assess overall national program 
effectiveness. 


11. Longer-term evaluation of the national program and the 
consortia should ask the basic questions: Did we 

achieve the development of an effective network? Did we 
provide meaningful space-related experiences for 
students? Did we achieve leverage from the seed 
funding? Did we achieve a genuine commitment from our 
universities, industry, NASA and other public agencies? 


12. We must avoid the development of an overly formalized 
and burdensome evaluation process, disproportionate 
to the programmatic size and level of effort. 


For all the reasons stated 
evaluations will remain to 
sagacity and judgment, and 
statistics to form a valid 
effect iveness . 


herein, ultimate assessment of 
some extent subjective, requiring 
an ability to look beyond 
gestalt assessment of program(s) 



